1. Field of the Invention
The present invention relates to computer programming, and deals more particularly with programmatically coordinating, and synchronizing execution of, automated testing in event-driven systems.
2. Description of the Related Art
Software testing for a complex application may require executing a large number of test cases to exercise sufficient coverage of paths through the software. In years past, human test personnel were largely responsible for carrying out the testing process, and in particular, for selecting which test cases should be executed, in which order, and under what conditions. A software test engineer would therefore invoke a test case, determine whether it completed successfully, and based on that determination, decide whether another test case should be invoked—or perhaps whether the test case should be re-executed after some type of software modification.
This level of human involvement in the testing process is very time-consuming and therefore costly. Accordingly, advancements have been made in automating the testing process. However, areas remain where automated testing has room for improvement. In particular, automated testing for systems that are event-based can be difficult. An event-driven system typically communicates with external and/or internal components through the use of events and listeners. Typically, listener software undergoes a registration process, whereby the listener informs the system of the event(s) in which it is interested; then, when a particular event is fired at run time, notification of the event occurrence is automatically sent to each registered listener. Listeners generally comprise application-specific code for reacting to the particular event(s) for which they register. Thus, upon notification of an event, the listener will perform certain action(s) pertaining to the event.
While automated test systems are known in the art, these systems are not sufficient for use with event-driven, listener-based systems. FIG. 1 illustrates problems that commonly arise, and will now be described.
The sample test case 100 in FIG. 1, referred to as “Testcase_1”, performs a series of steps which call several methods (“Method1” through “Method4”, in the example) sequentially. When this sample test case is started, Method1 in the event-driven system is invoked. See encircled numeral 1. Method1 (depicted symbolically at reference number 110) fires an event as it executes, where this event is referred to in FIG. 1 as “Event_A”. See reference number 111, where the invocation of Event_A is depicted as processing performed by the code of Method1, and reference number 112, representing the triggered event. In this example, Method1 executes within a first system named “SYSTEM 1” 120, and a listener 150 for Event_A is located in a second system named “SYSTEM 2” 160. Responsive to receiving notification of the firing of Event_A, listener 150 begins to execute its corresponding code 170.
Code 170 executes, performing processing for Event_A, and will complete in an unknown period of time. However, since Method1 is a non-blocking method in this example, it will finish and return control back to Testcase_1 immediately after Event_A is fired. See encircled numeral 2, where this is illustrated. Since Testcase_1 therefore believes that Method1 is finished, Testcase_1 will then proceed to invoke Method2. Method2 thus begins to execute while SYSTEM 2 is still processing Event_A. If Method2 is also non-blocking, then Testcase_1 will invoke Method3 immediately upon completion of Method2; similarly, Method4 may be invoked immediately following completion of Method3.
At some generally unpredictable time, the code 170 for processing Event_A reaches the end. As shown at 171, this sample listener code 170 triggers “Event_B” upon completion. Reference number 172 represents the triggered Event_B, and in this example, a listener 130 for Event_B is located in System_1 120. Responsive to receiving notification of the firing of Event_B, listener 130 begins to execute its corresponding code (which, for example, might comprise printing a result related to the processing of Event_A). When listener 130 completes its execution, the result of executing Method_1 is now available for Testcase_1, as shown generally by encircled numeral 3. However, it may happen that one, two, or even all three of the subsequent methods (i.e., Method2 through Method4) in the sample test case have already completed prior to completion of the event processing triggered by execution of Method1. Thus, the results from Method1 may be returned to an incorrect state (or at least an unpredictable state) of Testcase_1.
Furthermore, additional problems may result due to the unpredictable completion time of event processing that is triggered by Method1. For example, the processing of Method2, Method3, and/or Method4 may be affected by whether Method1 has completely finished. The unpredictable duration of the event processing may therefore cause one or more of these methods to be invoked during the wrong system state, such that the entire outcome of Testcase_1 becomes invalid.
One may think that an obvious solution is to insert intentional delay between the method calls to address this problem. See FIG. 2, where a revised version of the sample test case, now referred to as “New_Testcase_1” 200, is presented. In this revised version, a “wait” instruction 210 has been inserted between execution of Method1 and Method2, and is depicted in the example as waiting for “x minutes”. At first glance, this may seem like an acceptable solution. However, recall that the processing time of the events (Event_A and Event_B, in this example) is unknown. One would need to experiment with different time delays through trial and error for each method call. In addition, different computer systems might have different performance characteristics, such that the intended result is achieved on some systems but not on others. Furthermore, as time goes on, more functionality might be added to the systems under test, and each method might therefore take longer to run. As a result, one might have to repeatedly adjust the time delay in all existing test cases. In short, the intentional delay approach is error-prone and labor intensive, and is not a viable solution.
Accordingly, what is needed are techniques that avoid problems of the type described above, and which yield predictable results during automated testing of event-driven systems.